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ABSTRACT 

A vital aspect of performance measurement is the 
assurance that the service provided by the library or information 
service is valued by the users. Profiles of information use can help 
in the analysis of performance at a more detailed level than is 
normally practiced. The quality assurance toolkit described in this 
paper was designed to facilitate performance improvement for medical 
staff users of library and information services. The main sections of 
the toolkit cover assessment of user needs, information service 
provision, and focus on service outcomes. Each section follows a 
similar pattern: nature of evidence sought; supporting research 
evidence; and data required. Checklists of questions help identify 
the evidence that should be gathered, and toolkit users are directed 
to appropriate survey tools. The quality assurance toolkit is a set 
of guidelines and methods to help information professionals find 
evidence for the effectiveness, impact, and value of their service, 
and to use that evidence to find ways to improve the service. The 
evidence and targets in the toolkit relate to library service in the 
medical field, but many of the questions and methods can be adapted 
for any library and information service which serves distinct user 
groups and where benefits of use need to be assessed. (SWC) 
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Introduction 

A vital aspect of performance measurement is the 
assurance that the service provided by the 
library and information service is valued by the 
users. Many performance measurement tools 
attempt to gauge customer satisfaction but services 
that are ‘satisfactory’ may not in fact be providing a 
service that is truly useful as such satisfaction mea- 
sures depend on the expectations of the user which 
may be low. In the Value project (Urquhart and 
Hepworth, 1995 [a]) the value of a library and infor- 
mation service was measured in terms of outcomes 
of information use, ie. would the information make 
a difference to decisions made at work? As the users 
studied were clinicians (medical staff in hospitals 
and the community) clinical decision-making was 
the focus of the project. As clinical decisions made 
about patient care affect patient care outcomes the 
value of a library and information service, 
expressed in terms of benefits to clinical decision- 
making, can be related to organisational objectives 
for improved patient care. 

Study of the value placed on information and how 
that information is actually used provides a basis for 
an audit approach which is firmly based on the 
users, their information needs and patterns of infor- 
mation use. The quality assurance toolkit (Urquhart 
and Hepworth, 1995 [b]) which was derived from 
the findings of the Value project is aimed at perfor- 
mance improvement, not as a means in itself but for 
the clinician users of the library and information 
services. A more effective library and information 
service should thus contribute to more effective 
patient care. 

Tracking the Value of a Library and 
Information Service 

‘ . . .there is no point in deciding where your 
business is going rmtil you have actually 
decided with great clarity where you are now’ 

John Harvey Jones (1994) 



For clinician users the ‘business’ will cover infor- 
mation needs for patient care and also continuing 
education and research. Information for present 
patient care needs will usually be associated with 
immediate outcomes for patient care. However, 
information for present continuing education and 
research needs is less likely to be associated with 
immediate outcomes for patient care. Outcomes do 
have to be judged subjectively by the user and those 
outcomes concern present and future clinical deci- 
sion-making. 

In the Value project over 700 requests were 
tracked. These requests included interlibrary loan 
requests, end-user searches and mediated searches 
at 11 hospital sites in two regions of the UK. The 
hospital sites were chosen to give a good cross-sec- 
tion of types of hospital setting and library service. 
The clinicians were asked not just whether the 
information would be - or was - valuable to clinical 
decision-making, but also how it was valuable, and 
which categories of clinical decision-making would 
be affected. 

The results showed quite emphatically that infor- 
mation provided by NHS library and information 
services would or did affect clinical decision mak- 
ing. 89% of the clinician respondents agreed that the 
information obtained would affect one or more cate- 
gories of clinical decision-making. The question- 
naire survey was followed up by selective inter- 
views which provided case study evidence of the 
way information contributed to clinical decision- 
making. 

Profiles of information use can help in analysis of 
performance at a more detailed level than is normal- 
ly practised. The Value project showed that out- 
comes of use varied according to user group, cate- 
gorised by stage of postgraduate medical training. 
Career stage affects the outcomes of use in two 
ways. First, the stage affects the type of clinical 
decision-making priorities, and secondly, the career 
stage affects the gross valuation placed on the infor- 
mation provided. Not surprisingly, the most junior 
registered doctors value the information provided 
rSore as they are stiU on the steep slopes of the 
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learning curve. Another interesting finding was the 
high valuation placed on information by the group 
of remote users of MEDLINE, the users of the 
BMA (British Medical Association) Dial-Up MED- 
LINE service. Although this group is likely to be 
more information conscious (as they have made the 
effort to kit themselves out with the necessary hard- 
ware and software), the value of information on the 
desk-top, whether at home or in the office, is clearly 
appreciated. 

Evaluating Infrequent Use 



Toolkit Examples 

T he supporting research evidence included the 
following findings: 

- Research and publication purposes were 
involved to some extent in 55% of information 
service requests and searches. 

- Information needs and patterns of information 
use vary with the type of post held by the clini- 
cian and their career point. 

The relevant checklist questions include: 



A ny estimation of the performance or value of a 
library and information service should not 
neglect the infrequent user or the ‘non-user’. In the 
Value project, the surprising finding was the con- 
trast between the high valuation the junior doctors 
(senior house officers) placed on information pro- 
vided and with their low level of use of services that 
would provide up-to-date information on patient 
care. Analysis at this level of detail provides a better 
indication of performance failings than would be 
possible using gross statistics of use alone. In com- 
mon with other ‘inftequent user’ groups among 
clinicians, the senior house officers not only have 
problems accessing services, but also seem unaware 
of services that are available. 

Outcomes-based Audit 

T he analysis of the outcomes of information pro- 
vision, combined with a survey of the patterns 
of information need and use among a random sam- 
ple of clinicians (users and ‘non-users’ of the library 
and information services) provided the undeqjin- 
ning evidence for the quality assurance toolkit - 
having established that the information provided 
does have a value, that it does bring benefits, that 
knowledge gained can contribute to improvements 
in information provision. The audit cycle of needs - 
provision - outcomes is a circular process where the 
starting point can be at any stage. The main sections 
of the toolkit cover Assessment of user needs. 
Information service provision and Focus on the ser- 
vice outcomes (including targeting of services). 
Each of the topic sections follows a similar pattern: 
Nature of evidence to be sought (what to find out); 
Supporting research evidence (the reasons why evi- 
dence should be gathered) and Data required. 
Checklists of questions help to identify the evidence 
that should be gathered, and toolkit users are direct- 
ed to appropriate survey tools, based on those used 
and tested in the Value project 



- Are the following details obtained at registra- 
tion: grade of post, type of post (research, GP 
training etc.) contract term? 

- Would it be possible/useful to arrange a survey 
of the purposes of irrformation need among 
users and non-users (eg. to assess how well the 
service is performing in respect of purposes 
such as clinical care, education, research . . .)? 

If the answer to the latter question is ‘Yes’ the sug- 
gested method is a critical incident type survey 
which was used in the Value project. A one-page 
questionnaire was sent out once a week for four 
weeks to a random sample of clinicians asking them 
to think of one occasion during that week when they 
had needed information for patient care, teaching or 
continuing education. They were asked to tick cate- 
gories for purpose, sources tried and the degree of 
success obtained. The response rate for the Value 
project was 46% overall, and 69% of the sample 
replied at least once. 

Audit Survey 

T he Value project included an audit survey of 
around 35 UK libraries serving clinicians. This 
survey provided quantitative data (for a three-month 
period) on who was using the service, which ser- 
vices were being used, the profile of use for certain 
user groups, and the percentage of potential clini- 
cian users who did use the service. The main phase 
of the project had indicated the possible problem 
areas and the audit survey provided the quantitative 
data which indicated, for example, the level of use 
that might be expected for particular user groups. 
Although all the libraries surveyed collect statistics 
for aggregate use, very few could profile use by 
individuals and groups. 

The audit survey provided targets for levels of 
use of the library and information service for certain 
user groups and certain services. These should be 
regarded as initial benchmarks only, but they do 
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Urquhart and Hepworth: Deriving a QA Toolkit 
Figure 1 





TARGETS FOR ACTIVE USE 




Group or activity 


Medium site (1(X)-199 acute medical staff) 


Acute medical staff: percentage active use 


55%-65% (minimal) >70% (maximal) 


SHOs (senior house officers) - percentage active use 


60%-75% (minimal) >80% (maximal) 




RESULTS OF AUDIT SURVEY 




Group or activity 


Range found 


Acute medical staff: percentage active use 


40%-71% 


SHOs: percentage active use 


42%-96% 



provide indicators by which information services 
can assess their performance against that of other 
services. For example, see Figure 1 . 

Some library and information services do already 
target Senior House Officers (SHOs) and are 
rewarded in time by a percentage active use that is 
nearly 100%. Active use was defined for the pur- 
poses of the project as use of one (or more) of the 
following services: loans from stock, requests for 
material not held locally (interlibrary loans), CD- 
ROM end-user searches, and mediated searches. 
Borrowing one book in that three-month time period 
surveyed did make a user an ‘active user’. The audit 
survey complerpented study of those easily mea- 
sured services with an activity sampling exercise 
which helped to define the ‘browsing’ use of the 
service eg. consulting current journals, or reference 
books. 

Targeting Services 

R esources for library and information services 
are finite, and the problem today is to deal with 
real cuts. Better services for a neglected group of 
users may mean that other services have to be con- 
trolled or re-organised to make the most effective 
use of human and material resources. Librarians 
often operate in isolation and rigid mindsets about 
operations and procedures can deflect attention 
from required changes to meet alterations in user 
patterns. The checklist questions on targeting of ser- 
vices encourage reflection about the purposes of the 
service and the lateral thinking that may be required 
to provide a good service when budget constraints 
appear to limit the type of service deemed desirable. 
Checklist questions ask whether certain services can 
be costed, and whether benefits can be assessed not 



just for the user, but whether there are organisation- 
al benefits. Provision of a particular information 
service may give useful clues about the information 
behaviour of the user group, clues that will feed into 
improved provision of other services. There may 
also be training benefits for the information service 
staff providing these services. 

Sometimes services do have to be controlled or 
rationed in some way. Charging for some services is 
an emotive issue, but the alternatives, usually 
rationing or low-profile advertising of services, are 
often less fair on the occasional user, who might be 
the very person who needs to use the information 
service. It is therefore important for the library and 
information service to be able to answer checklist 
questions about trivial use of some services by an 
active minority. 

Resourcing and Performance 

hile services can be adjusted to match user 
needs, there is still a need for resources, and 
some librarians would argue that performance mea- 
surement is only worthwhile if the resources are 
sufficient. In the audit survey the funding levels for 
journals subscriptions were considered alongside 
the indicators for active use, to see whether poor 
resourcing would result in poor performance 
(judged by active use) and better than average 
resourcing would correspond to better than average 
performance (judged by active use). The results 
were mixed. Although better funding was generally 
associated with better performance there were some 
variations. Of the six libraries which had below 
average resourcing for journals, two were perform- 
ing better than might be expected for active use. At 
the other end of the scale, above average resourcing 
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did not guarantee an above average performance. 
Qearly the information professional in charge does 
make a difference, and can add value to resources. 

Conclusions 

T he toolkit is essentially a set of guidelines and 
methods to help information professionals find 
evidence for the effectiveness and impact of their 
service. There is no obligation to use the whole 
toolkit at once, or in any particular order. The aim 
was to produce a quality assurance toolkit which 
librarians could use themselves, creatively, to secure 
evidence that the information service does have an 
impact and value, and to use that evidence to find 
ways of improving the service. Certainly the evi- 
dence and the targets relate to libraries serving doc- 
tors but many of the questions and methods could 
be adapted for any library and information service 
which does serve distinct user groups and where 
benefits of use need to be assessed. The Value pro- 
ject is to be complemented by a study of the effec- 
tiveness of library and information supply to nurses, 
midwives and health visitors, and a similar toolkit 
will be produced. 
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